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SUMMARY: Participating in out-of-school time (OST) programs can 
be transformational in the lives of young people, but only if the 
efforts are high-quality. This guide provides recommendations to 
grantmakers about how they can increase quality in OST through 
local, regional, statewide and national grantmaking and other 
strategies. It was commissioned by Grantmakers for Education’s 
OST Funder Network as part of its Quality in Out-of-School Time 
Deep Dive Series for Grantmakers. 

DEFINITION: Grantmakers for Education’s OST Funder Network 
and this guide define “out-of-school time” to include all kinds of 
programs that happen outside of the classroom, before and after 
school, in the evenings, on weekends and during the summer; 
located in school buildings or community settings; managed or 
operated by schools, community organizations, parks, camps, faith- 
based organizations and other entities; and serving children and 
youth in grades K-12. This guide uses the terms afterschool and 
out-of-school time interchangeably. 


Grantmakers for Education (GFE) is a national network of hundreds of education philanthropies, united by a passion and commitment to improve 
public education and learning for all students of all ages, cradle to career. GFE is a force multiplier, harnessing the collective power of education 
grantmakers to increase momentum, impact, and outcomes for this nation's learners. We are proud to promote a culture of learning among 
education funders and provide a forum for interaction and engagement that builds upon and deepens the impact of our member's individual 
investments. Grantmakers for Education and its members believe in the power of what we can all achieve when we work together and learn from 
each other's successes and challenges. For more information or to learn about membership, please contact us at information@edfunders.org. 
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In recent years, OST 
programs have become 
the primary places where 
children can engage in 
the arts, physical activity, 
science and technology, 
leadership development 
and civic education - 
subjects and pursuits that 
have often been reduced 
during the school day as a 
result of increased focus on 
English language arts 
and mathematics. 


O ver the past several years, grantmakers focused on positive child and youth 
development and K-12 education reform have stepped up their investments 
in out-of-school time (OST) programs. This emphasis has coincided with a 
growing body of research connecting high-quality learning experiences 
during childhood and adolescence to the development of healthy and productive adults. 


Why are high-quality OST programs important? For starters, they typically engage youth 
in different ways from schools, homes, independent or peer group experiences. For 
example, in OST programs, youth often make independent choices about how and with 
whom to spend their time in a structured, safe and engaging learning context. FHigh- 
quality OST also offers young people the opportunity to forge positive relationships with 
adults and peers, and build social, cognitive and other vital skills. Research has shown 
that healthy relationships form the foundation of social/emotional skills and the ability to 
function successfully throughout life - at home, in school, with family and friends and as 
members of civic communities. The best OST programs epitomize the term "child/youth 
centered" in that the structures and activities are rooted in the interests and experiences, 
as well as the cultural and community context, of the participants and their families. 


In recent years, OST programs have also become the primary places where children can 
engage in the arts, physical activity, science and technology, leadership development 
and civic education - subjects and pursuits that have often been reduced during the 
school day as a result of increased focus on English language arts and mathematics. 
Furthermore, high-quality OST programs feature pedagogical strategies - such as 
project-based learning, inquiry-based investigations and student-driven pursuit of 
interests - that education experts recognize as the best route to fostering the engaged 
learners who will succeed in mastering not only academic content, but the complex 21st 
century skills necessary for lifelong success. 1 


As expectations that the OST field can deliver far more than what was expected of 
"school-age child care" 20 years ago have risen, so has program enrollment. According 
to the Afterschool Alliance America After 3PM 2014 data, nearly a quarter of all families 
(23%) currently have a child enrolled in an afterschool program, and participation in OST 
activities increased 60% from 2004 to 2014. 2 More than 10 million children now attend 
afterschool programs across the country. 
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GFE’s OST Funder 
Network members’ interest 
in program quality is 
growing: in 2013, 

63% reported they were 
funding evaluation and 
quality assessment at a 
systems level, up from 
47% in 2008. 


The stakes for successful OST programming, then, are high. But as OST becomes an 
increasingly crucial component in education reform and youth development, how will 
philanthropists, taxpayers and parents ensure that their investments produce desired 
youth outcomes? How will the field garner enough public and private support to serve 
the estimated 19.4 million children whose parents could not find an OST program in 2014? 
The answer is the same as the one recently noted by New York Times columnist David 
Kirp as he compared the results of various early education programs in an op-ed: The 
difference between poor and positive outcomes is "in a word, quality." 3 

In fact, the time is right for all OST funders to focus on quality. Research confirming the 
role that quality plays in driving regular participation and producing positive outcomes 
is definitive. Grantmakers have provided the support for this research, as well as for 
national field and system-building efforts at the state and local levels. These investments 
have resulted in the development of quality standards and quality improvement systems 
that include the use of assessment tools, access to data that drives changes in practice, 
professional development and other key resources. 

But many challenges remain to achieving equitable access to high-quality OST for all 
children across the country. While the infrastructure to enable major improvements in 
quality at scale is developing in many places, in others, such support is undeveloped or 
just beginning to emerge. All efforts face challenges in sustainably financing access for 
all children, regardless of age, socio-economic status or zip code. There is also a need 
for deep examination of the frameworks and tools currently used to define and measure 
quality to ensure they address the needs of diverse populations of young people, 
including those with special needs, English language learners, linguistic minorities, 
children of color and those from non-dominant cultural and ethnic communities. 

Since 2009, GFE's Out-of-School Time Funder Network has provided grantmakers 
with access to professional learning, dialogue, research and communications focused on 
improving the effectiveness of OST grant making. GFE's OST Funder Network members' 
interest in program quality is growing: in 2013, 63% reported they were funding 
evaluation and quality assessment at a systems level, up from 47% in 2008. With that 
in mind, in 2014 the network initiated a Deep Dive Series for Grantmakers focusing on 
OST quality. The Series has included web seminars, conference sessions, speakers and 
a podcast to enable funders to learn from each other and share lessons from current 
literature on OST program quality and grant making initiatives. See the Appendix 
for details on the Quality Series and the literature reviewed to inform the guide's 
development. 

This guide is intended to help funders of all sizes and locations focus on quality in 
their current and future OST-grant making. It reviews key pieces of the state-of-the-art 
knowledge about improving quality and the link between quality and youth outcomes. 
The guide also discusses efforts to define, measure and create systematic supports for 
continuous quality improvement in OST and also offers specific strategies for funders 
to consider - whether they invest locally, regionally or nationally, and in program 
operations, policy, research, system-building or other areas. Finally, the guide provides a 
set of recommended priorities for the field to catalyze significant change on the national 
and local levels, as well as case studies illustrating the different strategies grantmakers are 
pursuing to support OST quality improvement. 

Overall, this guide was produced to catalyze grantmakers to expand their attention to 
and investment in building quality - actions that will be critical to enabling all children 
and youth to have access to high-quality OST experiences that will lead to positive 
growth and development. 
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Background on Out-of-School Time Quality 


Defining Quality 

A seminal moment in the OST field occurred in 2001 when the National Research 
Council convened top researchers, practitioners and philanthropic leaders to form 
the Committee on Community-Level Programs for Youth. Supported by public and 
private funders, including the David and Lucile Packard Foundation, the William T. Grant 
Foundation and the Ford Foundation, the report published by the committee in 2002, 
Community Programs to Promote Youth Development, identified the following key 
features of programs that best support young people in developing positive personal 
and social assets: 

• physical and psychological safety and security; 

• structure that is developmentally appropriate, with clear expectations for behavior 
as well as increasing opportunities to make decisions, participate in governance 
and rule-making and take on leadership roles as one matures and gains more 
expertise; 

• emotional and moral support; 

• opportunities for adolescents to: 

• experience supportive adult relationships; 

• learn how to form close, durable human relationships with peers that support 
and reinforce healthy behaviors; 

• feel a sense of belonging and being valued; 

• develop positive social values and norms; 

• build and master skills; 

• develop confidence in one's abilities to master one's environment 
(a sense of personal efficacy); 

• make a contribution to one's community and to develop a sense of mattering 

• strong links between families, schools and broader community resources. 

The council's report was based on a comprehensive review of the best literature 
available about youth development programs, research and evaluation. 4 The report 
was instrumental in bringing attention to out-of-school time programs as an important 
focus for the health and well-being of children and youth and providing a foundation for 
research and public policy efforts. 5 The report's definition of the elements of program 
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quality most likely to lead to positive youth outcomes is still the most relevant and 
comprehensive definition used today by stakeholders of the OST field. 


The Importance of Quality- 

Supported in large part by philanthropic investments, research and evaluation 
efforts over the past two decades have built a knowledge base about the importance of 
quality in OST. Multiple studies have provided evidence that improving quality is more 
likely to get desired outcomes and draw parents and young people to the program. 

For example, a 2007 report examined the effects of high-quality after-school programs 
operating in high-poverty communities and found "higher standardized math test 
scores and better behavioral outcomes for students who regularly participated in high- 
quality after-school programs than for students who spent substantial after-school time 
without adult supervision. 6 A 201 1 meta-analysis of program evaluations completed by 
Joseph Durlak and Roger Weissburg concluded that higher quality programs resulted 
in successful academic, social, safety and familial impacts. Achieving high quality is 
important not only to producing positive benefits for participating youth, but also for 
securing increased and sustained public and private investments in OST programs. See 
Appendix A for a list of additional studies proving the link between quality afterschool 
programs and youth outcomes. 
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Components of a High Quality OST System 

The work conducted over the past two decades has helped the OST field make 
significant progress toward defining quality and proving its importance. With support 
from key grantmakers, several cities, regions and states are now building systems aimed 
at incorporating what is known about quality to improve the everyday OST experiences 
of young people. Such quality improvement systems include multiple components 
that are aligned and integrated, so they can address a broad swath of programs and are 
designed for sustainability over time. These foundational components include: quality 
standards, assessment tools, core competencies, professional development, workforce 
supports, data management and a lead intermediary. In addition, there are other key 
elements essential to building an infrastructure for achieving not only quality, but 
also sustained, equitable access to OST at scale: attention to providers' organizational 
capacity, partnerships with schools and other organizations, policy and advocacy, 
continued research and evaluation focused on effective policy and practices and 
capital/facilities support. 
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As of 2015, 33 states 
had adopted afterschool 
quality standards. 

A Wallace Foundation- 
commissioned report 
published in 2013 found that 
62% of cities coordinating 
afterschool programs use 
quality standards or a 
quality framework. 


The following list of components of a high-quality OST system was informed by the 
work of the Forum for Youth Investment, National Institute on Out-of-School Time, 
National Afterschool Association, Every Flour Counts, statewide afterschool networks, the 
Afterschool Alliance, the Flarvard Family Research Project and others. 

Quality Standards. A set of standards that clearly defines what quality looks like in 
an OST environment forms the foundation of a high-quality OST system. Increasingly, 
key stakeholders, including providers, public and private funders, technical assistance 
organizations, intermediaries, parents and others, are adopting and implementing 
such standards on the state, local and regional level. As of 2015, 33 states had adopted 
afterschool quality standards. A Wallace Foundation-commissioned report published in 
2013 found that 62% of cities coordinating afterschool programs use quality standards 
or a quality framework. 8 In addition to comprehensive standards, leading stakeholders 
have developed specific sets of standards focused on healthy eating and physical activity 
(by the National Institute on Out-of-School Time and National Afterschool Association), 
summer programs (by the National Summer Learning Association) and college and career 
readiness and STEM (by the Indiana Afterschool Association). 9 

Program Assessment Tools and Support for Continuous Quality Improvement. 

Quality standards are useful only if program providers employ them to assess and 
improve their programming. With that in mind, in recent years, grantmakers have funded 
the development of multiple evidence-based program assessment tools and training 
supports. 

For example, programs in Palm Beach County, Florida use a version of the David P. Weikart 
Center for Youth Program Quality's tool called the Youth Program Quality Assessment 
to assess how well their current practices align with regional quality standards. In 
Wyoming, providers are employing A Program Assessment System (APAS), developed 
by the National Institute on Out-of-School Time. Over the past three years, surveys 
completed by young people in the programs have shown improvements in attitudes, 
skills and behavior. (See Appendix B for more about how these evidence-based tools are 
used to improve quality). 10 

The 2009 Forum for Youth Investment brief, Measuring Youth Program Quality: A Guide to 
Assessment Tools, Second Edition, provides a useful overview of the 10 most commonly 
used quality assessment tools. 11 The report, published with support from the W. T. Grant 
Foundation, explains that programs need "tools that help concretize what effective 
practice looks like and allow practitioners to assess, reflect on and improve their 
programs." All of the tools included in the review measure six core constructs at varying 
levels of depth: relationships, environment, engagement, social norms, skill building 
opportunities and structures/routines. 


Since the second edition of the guide to assessment tools was published in 2009, 
grantmakers have supported the development of other tools in specific areas. For 
example, the Program in Education, Afterschool & Resiliency (PEAR) has developed 
tools specific to science, technology, engineering and math (STEM) programs, including 
Dimensions of Success and the Common Instrument suite. The Weikart Center is working 
with the National Summer Learning Association to develop an assessment tool geared to 
summer programs. (See Appendix B for details). 12 


Core Competencies. While standards focus on how programs must operate to 
achieve quality, core competencies define the skills and knowledge required of adult 
professionals to deliver high-quality programming. Core competencies provide a clear 
definition of effective youth development practice, serving as the basis for professional 
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development and training and other workforce supports. 13 Many localities have adopted 
the National Afterschool Association's core competencies in whole or tailored them to 
meet local needs. 

For example, in 2009, Prime Time Palm Beach County, a regional intermediary in Florida, 
convened local stakeholders to develop core competencies for practitioners. Prime 
Time produced a video module that provides an overview of the core competencies 
and takes practitioners step-by-step through the goal-setting process to plan their 
own professional development. (See Appendix B for more detail on how the core 
competencies fit into Palm Beach County's comprehensive quality improvement system). 


Professional Development and Workforce Support. Because one of the most 
important drivers of program quality is skilled staff, professional development and 
workforce support is critical to achieving quality improvement. That includes coaching, 
supervision and mentoring, in addition to training and formal education. Localities 
building OST professional development and workforce support use core competencies 
as the basis for designing both credit and non-credit bearing education pathways that 
consider the wide range of formal education and life experiences of OST staff. Significant 
expansion is needed, however, to bring these efforts to scale so more afterschool staff 
and youth workers have access to high quality pre- and in-service education and training 
that can lead to certificate and degree completion, career advancement and increased 
compensation. 


In addition to professional education, several communities and states are investing in 
other workforce supports, including: 

• career registries, providing OST professionals with the opportunity to collect, 
organize and showcase their education, training, and experience; 

• career lattices, assisting OST professionals in planning their career pathways by 
identifying and connecting to positions within the field and defining training and 
experience requirements; 

• compensation increases, targeted to OST professionals who demonstrate 
completion of various training pathways. 


Intermediary Organizations. Activities aimed at improving quality should not 
be the responsibility of each individual OST program or organization. Rather, a lead 
enterprise at the city, region or state level (referred to as an intermediary organization 
to differentiate it from a youth-serving organization) is often best positioned to take on 
the work of establishing a quality system in a locality. Its responsibilities may include 
setting guidelines for programs or organizations to take part in the system, identifying 
incentives and supports such as training and coaching, managing quality improvement 
data and coordinating the system components and partners. The Wallace Foundation 
has provided support to many cities for developing strong intermediaries, while the 
Every Flour Counts initiative has played a lead role in supporting a community of practice 
among lead intermediaries and providing technical assistance to cities and regions in 
developing intermediaries. 


Organizational Capacity. To ensure that quality improvements are sustained, many 
grantmakers interviewed for this guide pointed out that resources targeting the program 
level must be matched by resources bolstering capacity at the organizational level. In 
2008, a Wallace Foundation-commissioned study by Fiscal Management Associates 
found that "many [youth-serving] organizations lack the financial resources to invest 
significantly in administrative staff, facility needs, IT infrastructure and support and 
transformational purchases such as improved space." The study found that "the 
larger impact from working within this under-resourced administrative management 
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In addition to school 
partnerships, high-quality 
OST programs engage in 
partnerships with arts and 
cultural organizations, 
science centers and 
museums, institutions of 
higher education, private 
sector businesses and 
civic groups. 


environment is the limits it places on organizations' leaders' ability to be forward- 
looking and truly strategic." 14 A follow-up report released in 2015 outlines the design 
of a successful Wallace-backed initiative to equip organizations with the ability to plan 
and manage their financial resources. 15 In addition, leadership development is also a 
significant need facing the field. 

Partnerships with Schools and Other Organizations. Strong partnerships with 
schools and school districts enable OST programs to support students' academic success. 
Building relationships with teachers and administrators, participating in joint professional 
development, becoming familiar with individual teachers' and schools' curricula and 
teaching approaches and engaging in joint family outreach are a few strategies that 
schools and out-of-school programs can undertake to ensure that youth are able to build 
skills and engage in OST experiences that contribute to their school success. 16 

In addition to school partnerships, high-quality OST programs engage in partnerships 
with arts and cultural organizations, science centers and museums, institutions of higher 
education, private sector businesses and civic groups. Cross-sector partnerships provide 
mentors, tutors, new curricula and activity sequences, staff training opportunities, field 
trips, in-kind donations of technology and supplies and other resources that improve 
the quality of programs. Many OST intermediaries broker partnerships across the city, 
region or state to realize efficiencies of scale and streamline the administrative, outreach, 
coordination and training required to establish and maintain high-quality partnerships. 

Management Information Systems. In 2012, the National League of Cities' Institute 
for Youth, Education and Families produced a detailed guide to building OST-focused 
management information systems (MIS). Among the reasons cited for investing in an OST 
MIS were: 

1) "Providing policymakers and funders with accurate information on the utilization, 
quality and impact of afterschool programs to make better decisions and 
targeted investments at the systems level; 

2) Offering regular feedback to program managers and staff about the effectiveness 
of their efforts, both in absolute terms and relative to other programs, to promote 
continuous improvement; 

3) Reducing the time and money that programs spend completing paperwork 
to meet reporting requirements, freeing up valuable resources for direct 
programming with youth; and 

4) Empowering program sites and instructors with (near) real-time information on 
student outcomes such as attendance, behavior and academic performance that 
allows sites to tailor their instruction more closely to the needs of the youth they 
serve." 17 

At the same time, funders would be wise not to lose sight of the human element. For 
example, a 2015 Wallace-commissioned report by Chapin Flail at the University of Chicago 
examined the OST data management efforts of nine cities and found "for systems to be 
effective in their collection and use of data, they need to invest in more than just an MIS 
and related technology. Equally important are human capital - that is, a well-trained 
workforce with the skills and expertise to use the technology and interpret the data 
appropriately - and "social processes" (i.e. norms, routines, procedures and values) that 
encourage fruitful analysis of data." 18 

Research and Evaluation. Research and evaluation completed over the past two 
decades has created the foundation on which current OST quality improvement efforts 


GRANTMAKERS FOR EDUCATION | QUALITY IN OUT-OF-SCHOOL TIME 




are built. On a national level, continued research is critical to refine current tools and 
frameworks underlying quality improvement efforts, and to build our understanding of 
how young people learn. Key items on the national research agenda related to quality 
include effective and ongoing professional development models, the intersection 
between OST and social/emotional skill-building and defining effective practices for 
children's diverse needs, among many others. On the local and regional levels, many 
organizations are in the early stages of implementing assessment tools, creating data 
sharing agreements and building MIS solutions - efforts expected to yield useful data 
that can inform research on, and improvements in, the performance of OST systems. 

Policy, Advocacy and Communications. Ultimately, evidence of impact must 
be used to drive the public policy agenda toward providing sustainable financing 
allowing children to have equitable access to high-quality OST programs. The C.S. Mott 
Foundation-supported statewide afterschool networks - now in all 50 states - have led 
efforts to raise public awareness and engage policymakers in discussions and reform 
aimed at better supporting high-quality OST. They are joined by local and regionally- 
based intermediaries and national policy and advocacy organizations, including the 
Afterschool Alliance and others. Increasing the capacity of stakeholders at all levels to 
engage in policy development, advocacy and public will-building would help secure the 
substantial and sustainable policy and financing reforms needed to support quality at 
scale over the long term. 19 


For example, the David and Lucile Packard Foundation prioritized policy reform, 
advocacy, public awareness and communications in a seven-year effort to bring attention 
to and improve the quality of summer programs across California. The foundation's 
support launched the statewide Summer Matters campaign in 2010, aimed at increasing 
understanding among lawmakers, as well as school leaders, parents and others, about 
research findings and why investments in summer learning programs were critical for 
achieving expected student outcomes. See Appendix B for more details. 


Financing for Quality and Sustainability. Understanding the real costs of quality 
is key to building and sustaining a system over time, and particularly important to 
drive improved policy decisions. Consider a 2009 report from The Wallace Foundation 
on the cost of quality that aimed to equip decision-makers "to better assess different 
types of programs, their requirements and their associated costs, and weigh them more 
thoughtfully against the needs of their communities." The authors wrote, "We also hope 
the report opens the door to a more fact-based conversation about the costs of quality 
among policymakers who set reimbursement rates for OST programs, funders who want 
to ensure that their support more accurately matches their aims and OST providers who 
set priorities and create the budgets for their programs." The report found that paying 
for staff was the primary driver of cost overall, and costs varied significantly by program 
design, available resources and local conditions. An updated version of the report's cost 
calculator was released in February 2016. 20 


Capital and Facilities Improvements. The physical environment of an OST program 
is essential to quality, but providers often need capital or facilities improvements. 
Intermediaries can provide or coordinate assistance to providers in accessing grants 
or loans for needed improvements. Grantmakers whose strategies encompass capital 
assistance can also play a key role. 
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Getting to quality at scale in a locality will require attention to all of the elements 
described in the previous section - but, of course, a single grantmaker is unlikely to invest 
in every single one, nor are these one-time costs. This section includes suggestions for 
how grantmakers can focus on quality in their investments, whether their approach 
centers on providing operational or project dollars to youth-serving organizations, 
support for local or regional quality improvement systems or financing national level 
work. We differentiate these levels as follows: 

1. Organizational level. Funders providing operational and/or project 
support for youth-serving organizations can ensure that their grants support 
quality by setting expectations for organizations to meet quality standards, 
supporting them to use evidence-based program assessment tools and 
engage in continuous quality improvement and providing adequate 
resources for staff professional development. It is also important to invest in 
organizational capacity and leadership development. 

2. City/regional/state system-building level. Funders with the flexibility 
to support the creation and expansion of quality improvement systems can 
finance local, regional or state-level intermediaries both to facilitate use of a 
common set of program quality and youth outcome measures, and to help 
programs use resulting data to drive program quality improvement. With 
funder support, intermediaries can play a key role in putting all the elements 
of a quality OST system into operation. 

3. National level. Funders with a national reach can support networks 
of intermediaries and/or quality improvement efforts of national OST 
organizations. They can also support national intermediaries for research 
and evaluation into key challenges facing the field, public awareness, policy 
reform and advocacy on the federal level and dissemination of effective 
practices, among other strategies. 
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Getting to Quality at Scale: How Grantmakers Can 
Focus on Quality in Their Investments 



Funders can support networks of intermediaries, 
national OST organizations and national 
intermediaries to engage in: 



Research and evaluation of key challenges facing the field 


• Increased public awareness 

• Comprehensive policy and financing improvements 
at all levels of government 


Better dissemination of effective practices 




Funders can support local, regional or state- 
level intermediaries to work with providers 
in developing and promoting: 

• A common set of program quality standards 

• A common set of core competencies for staff 


Use of evidence-based program assessment tools to 
engage in continuous improvement 


Funders can support youth-serving organizations to: 


• Assess current practice against quality standards 

• Use evidence-based program assessment tools 

• Engage in continuous quality improvement 

• Promote staff access to high quality 
professional development 

• Strengthen organizational capacity 

• Prioritize leadership development 



ORGANIZATIONAL 
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Strategy Recommendations for Funders 


The following chart offers a range of strategies for funders to support quality at different levels - organizational, city/regional/ 
state and national. This chart is meant to demonstrate various possible entry points for those interested in program quality. The 
ideas here were informed in part by several grantmaker discussions convened by GFE's OST Funder Network and the National 
Afterschool Association's Funding Quality Initiatives, published as part of a series of focus briefs on the state of afterschool 
quality in 2014. 21 


Funding Strategies Chart 


COMPONENT 

ORGANIZATIONAL LEVEL 

CITY/REGIONAL/STATE LEVEL 

NATIONAL LEVEL 

QUALITY STANDARDS 

Require description of how 
program(s) align with local 
or statewide standards in 
proposals and grant reports. 

Support broad engagement 
of stakeholders and review 
of existing state and/or local 
standards. 

Support communications, 
engagement and advocacy 
around standards. 

Commission and disseminate 
current landscape of quality 
standards. 

PROGRAM ASSESSMENT 

TOOLS AND SUPPORT 

FOR CONTINUOUS 

QUALITY IMPROVEMENT 

Support grantees to use 
evidence-based assessment 
tools, ensuring alignment 
with what is already required 
by other funders. 

Fund coaching for grantees 
to use data for quality 
improvement. 

Support learning 
communities of grantees. 

Incentivize participation 
in quality improvement 
by providing programs 
with scholarships for staff 
professional development 
(PD), subsidies or resources 
for facility upgrades. 

Support intermediaries 
to offer incentives for 
participation and training, 
and ongoing, personalized 
assistance on using 
assessment tools. 

Support the development 
and refinement of quality 
assessment tools and 
approaches to continuous 
quality improvement. 

CORE COMPETENCIES 

Support programs to 
train staff in using core 
competencies to assess their 
own PD needs. 

Encourage/equip programs 
to incorporate core 
competencies into hiring 
practices. 

Support broad engagement 
of stakeholders and review of 
existing core competencies. 

Support communications, 
engagement and advocacy 
on core competencies. 

Support national 
organizations to disseminate 
and build awareness about 
core competencies and how 
they are used. 

Support examination of core 
competencies, refining as 
needed. 
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COMPONENT 


ORGANIZATIONAL LEVEL 


CITY/REGIONAL/STATE LEVEL 


NATIONAL LEVEL 


PROFESSIONAL 

DEVELOPMENT AND 

WORKFORCE SUPPORTS 

Support PD by funding 
release time, substitutes, 
tuition/fees and career 
advising. 

Require evaluations of PD. 

Support intermediaries to 
partner with practitioners, 
government, higher 
education and others to 

create non-credit and 
credit-bearing PD and 
establish other workforce 
supports including increased 
compensation programs, 
career registries and career 
lattices. 

Commission and disseminate 
reports on current landscape 
of PD and workforce supports 
nationally. 

Support OST leaders to 
explore national youth 
development certificate. 

Support online PD tied to 
quality standards and core 
competencies. 

INTERMEDIARY 

ORGANIZATIONS 


Use multi-year and/or general 
operating support to boost 
stability of intermediaries. 

Support strategic planning, 
board development, 
financial acuity, leadership 
development. 

Support national 
intermediaries with multiyear 
or general operating support. 

Support strategic planning, 
board development, 
financial acuity, leadership 
development. 

Commission and disseminate 
research into the optimal 
role for intermediaries and 
identify effective practices for 
intermediaries. 

ORGANIZATIONAL 

CAPACITY BUILDING 

Use multi-year and/or general 
operating support to boost 
stability of providers. 

Support strategic planning, 
board development, 
financial acuity leadership 
development. 

Create local cohorts of 
organizational leaders 
to learn from each other 
and further develop 
organizational leadership 
experience and skills. 

Support intermediaries to 
assess organizational capacity 
needs and develop sharable 
and scalable resources. 

Support policy, advocacy and 
public awareness actions to 
boost the health of the non- 
profit sector. 

Support policy, advocacy and 
public awareness actions to 
boost the health of the non- 
profit sector. 

PARTNERSHIPS 

Support program grantees 
to partner with schools, 
arts, cultural and civic 
organizations to enhance 
program quality. 

Support intermediaries to 
partner with arts, cultural, 
STEM-focused, civic, 
sports organizations and 
postsecondary education to 
offer new resources to OST 

programs. 

Support intermediaries to 
help providers build effective 
partnerships with school 
systems. 

Commission and 

disseminate research 
into effective partnership 
strategies; support national 
organizations to develop 
partnerships that benefit OST 
programs. 
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COMPONENT 


ORGANIZATIONAL LEVEL 


CITY/REGIONAL/STATE LEVEL 


NATIONAL LEVEL 


MANAGEMENT 

INFORMATION SYSTEMS 

(MIS) 

Ensure grantees have 
technology and training 
required to use an MIS and 
apply findings to quality 
improvement. 

Support the development 
of MIS to catalogue youth 
participation, program 
quality assessments, youth 
outcomes, parent/student 
satisfaction and other data. 

Support the development of 
online searchable databases 
of OST opportunities. 

Support dissemination of 
effective practices regarding 
MIS and using data to 
improve quality. 

RESEARCH AND 

EVALUATION 

Support program leaders 
and staff to learn about new 
research and apply lessons to 
practice. 

Support programs to evaluate 
their impact. 

Support intermediaries to 
keep program leaders up 
to date on new research, 
and identify and promote 
opportunities for local leaders 
to participate in research- 
practice exchanges. 

Commission and support 
research that expands the 
knowledge base around OST 
quality. 

Support research-practice 
exchanges. 

Fund research into quality 
and connections to practice 
for specific focus areas: 
social-emotional learning, 
STEM, literacy, serving English 
language learners, etc. 

Provide co-funding matches 
to programs and initiatives 
that are awarded federal 
grants. 

POLICY, ADVOCACY AND 

COMMUNICATIONS 

Support practitioners to 
build skills and participate 
in advocacy and public 
awareness for OST. 

Support intermediaries to 
train and assist practitioners 
and lead advocacy and public 
awareness for OST. 

Support intermediaries to 
advocate alignment of quality 
improvement resources from 
21st Century Community 
Learning Centers and child 
care funding. 

Support intermediaries to 
train and assist practitioners 
and lead advocacy and public 
awareness for OST. 

Support intermediaries to 
advocate alignment of quality 
improvement resources from 
21st Century Community 
Learning Center and child 
care funding. 

Lead conversations on role 
of high-quality OST in K-12 
education. 

FINANCING FOR 

QUALITY AND 

SUSTAINABILITY 

Understand the cost of 
quality and support providers 
with grants adequate to 
operate quality programs. 

Support intermediaries to 
raise broad awareness about 
what quality OST costs and 
to train providers on building 
sustainable and diversified 

revenue streams. 

Commission and disseminate 
reports that analyze and 
document the cost of 
operating quality OST 
programs. 

CAPITAL AND FACILITIES 

SUPPORT 

Support capital or facilities 
improvements to offer 
better-quality program 
environments. 

Support intermediaries to 
assist providers in accessing 
resources for capital 
improvements. 

Commission and disseminate 
publications on how 
providers can access capital/ 
facility improvement support. 
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Through a review of key literature in the field and in-person meetings with grantmakers 
conducted to develop this guide, several common principles underlying funding OST 
quality improvement emerged: 

Every funder has a role to play. Whether a grantmaker has a local, regional or national 
footprint or a priority to support such areas as program operations, system-building, 
research, policy change or professional development, every funder should make 
quality a part of its goal. To that end, a grantmaker providing only program grants for 
direct services to youth, can (1) require and provide resources for each grantee to use a 
validated quality assessment tool and (2) ensure that program budgets have sufficient 
resources for organizational leaders and staff to engage in quality improvement. 

Grantmakers can also provide resources for their OST program grantees to engage in the 
quality improvement process as a learning community, with professional coaching and 
support. For example, the New York City-based Youth Development Institute facilitates 
a peer-driven network of 40 career internship program providers focused on quality 
improvement. Professional development for staff of member organizations includes 
everything from workshops, structured site visits and on-site technical assistance to 
a listserv where members notify the group about activities other program's interns 
can attend, as well as exchange ideas and program advice, and build community. See 
Appendix B for a more in-depth examination of one funder's investment. 

Achieving quality improvement at scale requires investments at the system 
level. While supporting individual or groups of grantees is helpful, such a strategy, of 
course, is ultimately limited in impact to those organizations. To build quality at scale 
within their geographic footprint, grantmakers who have the flexibility to support strong 
intermediaries as anchors of a quality improvement system should consider doing so. For 
more examples of systemic support, see the case studies focused on Palm Beach County, 
Florida, and the states of California and Wyoming in Appendix B. 

A focus on equity is crucial and will require differentiation of support. Pedro 
Noguera of UCLA recently said, "Access to high-quality afterschool programs could 
play a decisive role in reducing educational disparities, but this will only occur if we 
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“Funders should stop 
investing in each 
program creating its own 
evaluation or quality 
assessment measures. 
Instead, funders 
should support the 
dissemination of tools we 
have in common so we 
can grow the evidence 
base and grow the held.” 

Grantmakers and Thought Leaders 
on Out-of-School Time 


remain vigilant in advocating for equity in the sector." 22 Increasing equity will require 
grantmakers to provide organizations working in low-income communities or serving 
high-needs children a different and more extensive set of supports to achieve quality. 
Those efforts might include, for example, assistance in building organizational capacity 
and sustainability over the long term, helping staff and leaders get access to training, 
providing resources for facilities improvement, transportation subsidies, access to 
comprehensive services for low-income children and families and other support. In 
addition, more research is needed to determine how quality frameworks should be 
differentiated to meet the diverse needs of all youth. 

Quality improvement requires long-term investments. One common challenge 
for OST organizations working on quality improvements is that funders do not routinely 
take into account the timeframe and costs associated with higher quality programming. 
Improving OST quality is not a discrete project that can be initiated, implemented, 
evaluated and concluded. Instead, achieving substantial change will require grantmakers 
to set up a different way of doing business. Funders should consider both multi-year and 
general operating grants to increase organizational stability and free more management 
time and attention to focus on program support and staff supervision. Grantees need 
ongoing assistance to assess quality, implement improvements and incorporate 
improvement cycles into their routine work. Staff turnover in many OST programs 
intensifies the need for ongoing training (whether for new staff or veterans to hone their 
skills) and may undermine the organization's ability to institutionalize quality practices. 

Don't reinvent the wheel - or fund a grantee to do so. As one thought leader noted 
in the OST Funder Network 2014 report Grantmakers and Thought Leaders on Out-of- 
School Time, "Funders should stop investing in each program creating its own evaluation 
or quality assessment measures. Instead, funders should support the dissemination 
of tools we have in common so we can grow the evidence base and grow the field." 23 
Numerous organizations have designed and implemented quality standards, assessment 
tools, data management systems, career lattices and registries and credentialed training 
pathways. While it may not be useful to apply each existing tool or system in all places, 
chances are there is an example up and running that can be adapted to a program's 
needs. Successful adaptation by funders and grantees entails taking into account the 
different programs, assets and challenges in various communities, regions and states. 

Both financial support and provider motivation are needed to have impact at 
scale. Funder or policymaker mandates requiring quality improvement will not singularly 
boost outcomes. Nor will the best intentions of providers to improve quality result in 
change without sufficient knowledge, resources and support. Rather, the best quality 
improvement strategies are characterized by partnerships among various stakeholders in 
a community or at the regional, state or national level. 

Grantmakers should understand the quality landscape. It is critical that funders 
understand what efforts are already underway in the targeted geography or focus 
area to improve quality at the program, organization and/or system level. For example, 
if providers are already required to use a particular quality assessment tool, insisting 
on another is counterproductive. Grantmakers should instead identify the gaps in the 
existing work. In addition, while any one funder does not have to support all parts 
of quality work, there will likely be limited success or impact if all elements are not 
addressed. Well-informed funders working together to support high-quality programs 
should be the rule, not the exception. 
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Chapter two illustrated ways in which grantmakers can prioritize quality improvement 
in their out-of-school time investments, regardless of portfolio size, investment strategy 
or geographic footprint. This chapter recommends priorities for an action agenda for 
the field. Collaborative leadership by grantmakers on these priorities has the potential 
to catalyze impact on a national scale, which will, in turn, increase the opportunities for 
more young people to experience high-quality OST. 


Knowledge Building/Research Priorities 

To improve quality at scale, it is critical for funders to take full advantage of the 
knowledge gained through existing efforts. Below are examples of knowledge building 
and research priorities that lend themselves well to aligned and/or collaborative 
grantmaker support: 

Better Understanding the Landscape. The field lacks a comprehensive landscape 
of community, regional and state-level quality improvement systems. An all-inclusive 
"state-of-the-field" report could answer key questions for a range of stakeholders: What 
states and cities have developed which components? What processes did they use? 

How are the components similar or different? How well is the component working and 
what is the evaluation methodology? Which strategies are most effective in scaling 
quality improvement resources to all programs in a community? Such an analysis could 
be used to share lessons across organizations and localities and inform local and state 
policymakers interested in improving OST quality. An online database searchable by 
geography and quality component would enable stakeholders to gain access to detailed 
information. 

Similar mapping of professional development and skill-building initiatives could provide 
a detailed digest and analysis of the successes and challenges of various credential 
programs at postsecondary education institutions and community-based training 
organizations, as well as the potential for creating a nationally recognized youth 
development credential. Finally, up-to-date workforce studies detailing demographics, 
job categories and relevant salaries, career pathways, education requirements and 
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One common challenge 
for OST organizations 
working on quality 
improvements is that 
public and private 
funding falls short of 
the costs required to 
achieve high-quality 
programming. 


other factors would provide critical information for the design, piloting and scaling of 
new approaches to professional development and skill-building that are responsive to 
workers' needs. 

Clarifying What Quality Costs. One common challenge for OST organizations 
working on quality improvements is that public and private funding falls short of the 
costs required to achieve high-quality programming. Public/Private Ventures and The 
Finance Project published a Wallace Foundation-supported study in 2009 that analyzed 
and helped clarify costs for quality out-of-school time programs based on data from 
111 programs in six cities. 24 In 2016, The Wallace Foundation released an updated cost 
calculator that reflects both the change in the general cost of living nationally and 
changes in the relative cost of living across cities. There are still important questions to be 
answered, including, as mentioned in the original report, "how cost components vary for 
different auspices and different populations; opportunities to realize economies of scale 
in program operations; and effective financing strategies for OST programs." The field 
could benefit from deep exploration of these questions. 

Building the Evidence Base. Grantmakers should continue to help the field build the 
evidence base of what program strategies work successfully and why. Key items on the 
national research agenda related to quality include effective and ongoing professional 
development models, refining current tools and frameworks, the intersection between 
OST and social/emotional skill-building, STEM and other areas and defining effective 
practices for children's diverse needs, among many others. One strategy is to provide the 
match for such federally funded awards as Investing in Innovation and other evidence- 
focused programs. 


Policy and Advocacy Priorities 

For long-term sustainability, funders need to engage the public and policymakers in 
building support for policies and financing that enable providers to deliver high-quality 
programs. The mapping and research outlined above will help stakeholders advocate 
in local, state and federal policy arenas. In addition to providing support for policy and 
advocacy work, grantmakers should pay attention to the potential in aligning multiple 
public funding sources' quality improvement resources, particularly 21st Century 
Community Learning Centers and Child Care Development Block Grant programs at the 
state level. Grantmakers could support the development of a report on this alignment or 
lack thereof, along with follow-up activities enabling states that have achieved progress 
in aligning these and other quality improvement efforts to share effective strategies. 


Priorities to Improve Program Practice 

Focusing on Equity. In the development of this guide, leading stakeholders expressed 
a desire to work collaboratively with their peers to examine existing quality constructs, 
tools and professional development to determine the extent that they prepare OST 
programs and professionals to meet the diverse needs of children - including those 
with disabilities, children of color, children of varied cultural and linguistic backgrounds, 
immigrant children and English language learners. There is an increasing sense that 
"one size does not fit all" when it comes to OST quality and the field could target its 
approaches to achieve greater impact and effectiveness for youth. Grantmakers could 
help by sponsoring a series of conversations among lead stakeholders, including the 
important step of identifying diverse voices to come to the table. An important question 
for grantmakers is: Are resources going to where they are most needed? Funders 
committed to equity have an obligation to understand how/if their current investments 
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promote that goal and look for ways to alter their financing practices in order to assist 
organizations serving high-need populations trying to provide quality services. 

Extending Quality Improvement Efforts of National Youth Development 
Organizations. National youth development organizations (YDOs) reach millions of 
young people every year, and in many locations are the key providers of OST programs. 
There is an opportunity for grantmakers to bolster quality of programming by national 
YDOs and, at the same time, incentivize them to work in collaboration with other 
organizations in their local communities on quality initiatives. For example, the Houston 
YMCA led the development of a Quality Improvement System for YMCA and other 
local OST programs. The organization is also working with its national office to share 
lessons learned from this effort throughout the YMCA system across the country. Further 
exploration could include the possibility of larger, more established organizations 
assisting smaller OST programs with back office support to help them spend scarce 
dollars on improving quality, as well as setting up more knowledge exchanges among 
organizations of different sizes and strengths to share effective practices. 


Developing Adaptable Credentials for Youth Professionals. Grantmakers could also 
support a multi-sector group of youth development leaders, trainers and representatives 
from intermediaries and postsecondary education to develop adaptable models for 
youth worker and afterschool credentials, such as certificates, associate or bachelor's 
degree programs. 
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I n the last 15 years, grantmakers have made significant contributions to improving 
the quality of OST programs and systems through a range of investments at the 
national, regional, state and local levels. These contributions have resulted in much 
better understanding about what quality looks like and what it takes to get there. 
Now the OST field faces the triple challenge of: 


1) improving quality at scale as OST enrollment increases; 

2) building the capacity of many more afterschool staff to implement and manage 
high-quality youth programs effectively, and 

3) developing sufficient and sustainable resources to ensure ongoing quality. 


There is a need for grantmakers to stay at the table, step up their investments and 
prioritize quality as a non-negotiable component of their efforts. Philanthropic leadership 
will be critical to meet the challenge of providing all youth the chance to attend high- 
quality OST programs throughout their childhood and adolescence. Such actions 
can ensure that young people not only stay safe and sound, but also are inspired and 
challenged to broaden their horizons and reach their potential. 
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APPENDIX A - Quality Resources 


This resource list includes links to key documents related to quality in out-of-school time, however it is not meant to be an exhaustive 
digest of the topic. 

An excellent overview piece is the 2013 compendium produced by the Expanded Learning and Afterschool Project: Expanding 
Minds and Opportunities: Leveraging the Power of Afterschool and Summer Learning for Student Success. This document, which 
provides a comprehensive overview of afterschool and summer programming, with examples of effective practices, programs and 
partnerships, is edited by Terry Peterson and funded by the Charles Stewart Mott Foundation. Link: http://www.expandinglearning.org/ 
expandingminds/ 

In addition, The Wallace Foundation maintains an extensive digest of publications on its Knowledge Center, including afterschool, 
summer and expanded learning. Link: http://www.wallacefoundation.org/knowledge-center/Pages/default.aspx 


EVALUATION AND OUTCOMES 

Taking a Deeper Dive into Afterschool: Positive Outcomes and Promising Practices (2014) 

Author: Afterschool Alliance 

Funder: Walton Family Foundation 

Link: http://afterschoolalliance.org/documents/Deeper__Dive_into_Afterschool.pdf 

A Meta-Analysis of After-School Programs that Seek to Promote Personal and Social Skills in Children and Adolescents (2010) 

Authors: Durlak, J.A., Weissberg, R.P., and Pachan, M.K. 

Funder: William T. Grant Foundation 

Link: Research brief: http://www.expandinglearning.org/docs/Durlak%26Weissberg_Final.pdf 

Full paper published in the American Journal of Community Psychology available for a fee here: http://link. 
springer.com/article/1 0. 1 007/s 10464-010-9300-6 

Outcomes Linked to High-Quality Afterschool Programs: Longitudinal Findings from the Study of Promising Afterschool 
Programs (2007) 

Authors: Vandell, D. L., Reisner, E. R., and Pierce, K. M. 

Funder: Charles Stewart Mott Foundation 

Link: http://education.uci.edu/childcare/pdf/afterschool/PP%20Longitudinal%20Findings%20Final%20Report.pdf 

The Impact of After-School Programs that Promote Personal and Social Skills (2007) 

Authors: Durlak, J. A. and Weissberg, R. P. 

Funder: William T. Grant Foundation 

Link: http://www.uwex.edu/ces/4h/afterschool/partnerships/documents/ASP-Full.pdf 

Community Programs to Promote Youth Development (2002) 

Author: Committee on Community-Level Programs for Youth, Board on Children, Youth, and Families, Division of Behavioral 

and Social Sciences and Education, National Research Council, and Institute of Medicine 
Funders: Ford Foundation, the David and Lucile Packard Foundation, William T. Grant Foundation, Carnegie Corporation of 

New York, U.S. Departments of Health and Fluman Services, blousing and Urban Development, and Justice. 

Link: http://www.nap.edu/catalog/10022/community-programs-to-promote-youth-development 


OST SYSTEM BUILDING 

Growing Together, Learning Together, What Cities Have Discovered About Building Afterschool Systems (2015) 

Author: Browne, D. 

Funder: The Wallace Foundation 

Link: http://www.wallacefoundation.org/knowledge-center/after-school/coordinating-after-school-resources/ 

Documents/Growing-Together-Learning-Together.pdf 
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Funding Quality Initiatives (2014) 

Author: National Afterschool Association 

Link: http://c.ymcdn.com/sites/mnaeyc-mnsaca.org/resource/resmgr/Docs/NAA4MNA.pdf 

Is Citywide Afterschool Coordination Going Nationwide? An Exploratory Study in Large Cities (2013) 

Author: Simkin, L. et a!., FHI 360 

Funder: The Wallace Foundation 

Link: http://www.wallacefoundation.org/knowledge-center/after-school/coordinating-after-school-resources/ 

Pages/City wide-Afterschool-Coordination-Going-Nationwide-An-Exploratory-Study-in-Large-Cities.aspx 

Building City wide Systems for Quality: A Guide and Case Studies for Afterschool Leaders (2012) 

Author: Yohalem, N., Devaney, E., Smith, C. and Wilson-Ahlstrom, A., the Forum for Youth Investment 

Funder: The Wallace Foundation 

Link: http://forumfyi.org/building__system_quality 

Seminar: http://www.wallacefoundation.org/view-latest-news/events-and-presentations/Pages/Webinar-lmproving- 

Quality-Systemwide.aspx 

Building Management Information Systems to Coordinate City wide Afterschool Programs: A Toolkit for Cities (2012) 

Author: Kingsley, C., National League of Cities Institute for Youth, Education, and Families 

Funder: Annie E. Casey Foundation and The Wallace Foundation 

Link: http://www.wallacefoundation.org/knowledge-center/after-school/collecting-and-using-data/Pages/ 

Building-Management-Inf ormation-Systems-to-Coordinate-Citywide-Afterschool-Programs.aspx 


PROGRAM ASSESSMENT AND QUALITY IMPROVEMENT TOOLS AND GUIDES 
Quality Improvement Tools 

• David P. Weikart Center on Youth Program Quality, see: http://www.cypq.org 

• Program in Education, Afterschool and Resiliency (PEAR), see http://www.pearweb.org/ 

• National Institute on Out-of-School Time, see http://www.niost.org 

Measuring Youth Program Quality: A Guide to Assessment Tools, Second Edition (2009) 

Author: Yohalem, N., Devaney, E., Smith, C. and Wilson-Ahlstrom, A. with Fischer, S. and Shinn, M., the Forum for Youth 

Investment 

Funder: William T. Grant Foundation 

Link: http://forumfyi.org/content/measuring-youth-program-quality-guide-assessment-tools-2nd-edition 

Structuring Out-Of-School Time to Improve Academic Achievement: A Practice Guide (2009) 

Authors: Beckett, M., Borman, G., Capizzano, J., Parsley, D., Ross, S., Schirm, A., and Taylor, J. National Center for Education 

Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education 
Funder: U.S. Department of Education 

Link: http://ies.ed.gov/ncee/wwc/PracticeGuide.aspx?sid=10 

A Practitioner's Guide: Building and Managing Quality Afterschool Programs (2009) 

Editors: Jordan, C., Parker, J., Donnelly, D. and Rudo, Z., SEDL 

Funder: Charles Stewart Mott Foundation 

Link: http://www.sedl.org/afterschool/practitioners_guide_to_afterschool_programs.pdf 

After-School Toolkit: Tips, Techniques and Templates for Improving Program Quality (2008) 

Authors: Bradshaw, M., Furano, K. and Gutierrez, N., Public/Private Ventures 

Funder: James Irvine Foundation 

Link: http://ppv.issuelab.org/resource/after_school_toolkit_tips_techniques_and_templates_for_improving_ 

program_quality 
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Putting It All Together: Guiding Principles for Quality Afterschool Programs Serving Preteens (2008) 

Author: Public/Private Ventures 

Funder: Lucile Packard Foundation for Children's Health 

Link: http://www.statewideafterschoolnetworks.net/putting-it-all-together-guiding-principles-afterschool- 

programs-serving-preteens 


CORE COMPETENCIES FOR OST PROFESSIONALS 

Youth Work Core Competencies: A Review of Existing Frameworks and Purposes (2009) 

Authors: Starr, B., Yohalem, N. and Gannett, E. 

Funder: School's Out Washington 

Link: http://www.niost.org/pdf/Core_Competencies_Review_October_2009.pdf 


ORGANIZATIONAL CAPACITY BUILDING 

The Skills to Pay the Bills: An Evaluation of an Effort to Help Nonprofits Manage Their Finances (2015) 

Authors: MDRC and Child Trends 

Funder: The Wallace Foundation 

Link: http://www.wallacefoundation.org/knowledge-center/after-school/financial-management-for-nonprofits/ 

Pages/The-Skills-to-Pay-the-Bills.aspx 

Administrative Management Capacity in Out-of-School Time Organizations: An Exploratory Study (2008) 

Authors: Summers, J. and Price, L., Fiscal Management Associates 

Funder: The Wallace Foundation 

Link: http://www.wallacefoundation.org/knowledge-center/after-school/financial-management-for-nonprofits/ 

Pages/ Administrative-Management-Capacity-in-Out-of-School-Time-Organizations.aspx 


FINANCING OUT-OF-SCHOOL TIME PROGRAMS 

Tough Times, Tough Choices in After-School Funding: Pathways to Protecting Quality (2012) 

Author: McCombs, J., Nataraj Kirby, S. and Joseph Cordes, RAND Corporation 

Funder: The Wallace Foundation 

Link: http://www.wallacefoundation.org/knowledge-center/after-school/quality-and-cost/Pages/Tough-Times- 

Tough-Choices-in-After-school-Funding.aspx 

The Cost of Quality Out-of-School-Time Programs (2009) 

Authors: Baldwin Grossman, J. and Lind, C. et al., Public/Private Ventures, the Finance Project 

Funder: The Wallace Foundation 

Link: http://www.wallacefoundation.org/knowledge-center/after-school/key-research/Pages/The-Cost-of- 

Quality-of-Out-of-School-Time-Programs.aspx 

For the updated cost calculator, see: http://www.wallacefoundation.org/cost-of-quality/Pages/default.aspx 


POLICY, ADVOCACY AND PUBLIC AWARENESS 

Strengthening Partnerships and Building Public Will for Out-of-School Time Programs (2010) 

Authors: Clapp Padgett, H., Deich, S. and Russell, L., National League of Cities Institute for Youth, Education, and Families. 

Funder: The Wallace Foundation 

Link: http://www.wallacefoundation.org/knowledge-center/after-school/coordinating-after-school-resources/ 

Pages/Strengthening-Partnerships-Building-Public-Will-Out-of-School-Time.aspx 
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Out-of-School-Time Policy Commentary #13: Speaking in One Voice: Toward Common Measures for OST Programs & Services (2008) 

Authors: Wilson-Ahlstrom, A., Yohalem, N. and Donner, J., the Forum for Youth Investment, the Collaborative for Building 

After-School Systems 
Funder: William T. Grant Foundation 

Link: http://forumfyi.org/files/OSTPC13.pdf 


QUALITY IN CONTENT FOCUS AREAS 

Literacy 

Building Literacy in After School (2015) 

Author: Afterschool Alliance 

Funder: Dollar General Literacy Foundation 

Link: http://afterschoolalliance.Org//documents/issue_briefs/issue_building_literacy_67.pdf 


Science, Technology, Engineering and Math (STEM) 

Click2sciencepd.org: an interactive, professional development site for trainers, coaches, site directors and frontline staff/volunteers 
working in out-of-school time STEM programs, serving children and youth. 

Funder: The Noyce Foundation 

Link: http://www. click2sciencepd.org 

STEM in Afterschool System Building Toolkit 

Author: Mainspring Consulting and Project LIFTOFF, informed by strategies and tools of the Charles Stewart Mott 

Foundation-funded Statewide Afterschool Networks 
Funder: The Noyce Foundation 

Link: http://www.expandingstemlearning.org 

Examining the Impact of Afterschool STEM Programs (2014) 

Authors: Krishnamurthi, A., Ballard, M., and Noam, G. 

Funder: The Noyce Foundation 

Link: http://afterschoolalliance.org/ExaminingthelmpactofAfterschoolSTEMPrograms.pdf 


Social and Emotional Learning 

Preparing Youth to Thrive: Promising Practices in Social Emotional Learning (2016) 

Authors: Smith, C., McGovern, G., Larson, R., Hillaker, B. and Peck., S.C. 
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Appendix B: Case Studies 


Creating OST Quality Improvement and 
Professional Development at Children’s 
Services Council of Palm Beach County 

The story of Prime Time Palm Beach County's quality 
improvement efforts is one of methodical, long-term 
commitment. 

Created in 2000, Prime Time Palm Beach County is an 
independent nonprofit intermediary organization focused 
on building the capacity of the county's afterschool 
programs to offer high-quality learning that keeps youth 
engaged and on track for academic success and healthy 
social development. Its service area. Palm Beach County, 
Florida, is home to more than a quarter million children 
under age 18, with an estimated 19.4% living below poverty 
level. The organization works with 250+ afterschool sites 
and 1,700 practitioners, ultimately impacting 20,000 local 
youth each year. 

Prime Time's largest single funder, the Children's Services 
Council of Palm Beach County, was created in 1986 as a 
result of a voter-approved property tax levy directed to 
support programs that improve the lives of children and 
their families. The levy was reapproved by 85% of voters 
in 2014. 

The organization's approach to improving program quality 
began in 2002 with a convening of representatives from the 
school district, the county parks and recreation department, 
afterschool providers, the local state college and funders to 
define quality standards for Palm Beach County afterschool 
programs. The resulting standards, along with the Palm 
Beach County Afterschool Core Competencies for staff 
developed by Prime Time and other stakeholders in 2009, 
form the foundation of Prime Time's current Quality 
Improvement System (QIS). 

As of fall 2015, 136 programs have voluntarily adopted the 
quality standards and are actively participating in QIS (there 
may be multiple programs per organization). Entry is by 
application, and minimum requirements include dedicated 
program space, 20 or more youth attending daily, and 
up-to-date compliance with state licensure status. Once 
an afterschool program is accepted into QIS, a full range of 
targeted professional development supports are available 
to its staff. These include scholarships to attend college 
courses, career advising to navigate educational pathways 
and the college system, access to a registry system to store 


educational and professional accomplishments and financial 
incentives to reward practitioners as they progress and 
complete specific milestones. Afterschool staff also receive 
individual coaching on youth development strategies that 
support program improvements. 

A detailed description of the components of Prime Time's 
QIS follows: 

• Measurement tools. Programs use the Palm 
Beach County Program Quality Assessment (PBC- 
PQA) measurement tool to understand their level 
of adherence to the standards. The PBC-PQA is a 
customized quality assessment tool based on the 
Youth Program Quality Assessment. 

• Quality advisor. Programs work with a quality advisor, 
employed by Prime Time, who offers consultation, 
coaching and technical assistance for continuous 
quality improvement based on PBC-PQA results. 

In 2015, Prime Time launched a new QIS Incentives 
Program, which will provide financial awards to 
programs based on performance, to re-invest back 
into the program. 

• Career advisor. All OST practitioners in the county 
- not just those in programs participating in Prime 
Time's QIS system - have access to a career advisor. 
These advisors help staff to participate in professional 
development according to individual strengths and 
areas of growth mapped to core competencies. Career 
advisors help practitioners decide which educational 
pathway - credit or non-credit - is best for them, assist 
with planning and scheduling classes or trainings 
and connect them to financial resources, such as 
scholarships and salary supplements. 

• Scholarship support. Prime Time provides 
scholarship support to practitioners. Operated in 
partnership with Palm Beach State College, the 
credit-bearing pathway leads to a Youth Development 
College Credit Certificate, an Associate in Science 
Degree in Human Services (Youth Development 
Concentration), and a Bachelor of Applied Science 
Degree in Supervision and Management. The non- 
credit bearing pathway awards practitioners earn 
Continuing Education Units leading to the 40-Hour 
School Age Certification, with options to earn Director 
Credentials at initial and advanced levels. Since 2007, 
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Prime Time has awarded more than 1,500 scholarships 
to afterschool professional staff for college course 
work, professional credentialing and to attend 
conferences. 

• Salary supplements. Through the ACHIEVE OST 
program and the Children's Services Council, Prime 
Time offers salary supplements to QIS practitioners 
earning less than $17.50/hour who successfully 
complete milestones along their professional 
development path. The program is designed to 
increase afterschool program stability and improve 
program quality by reducing staff turnover and 
encouraging afterschool practitioners to continue 
their education. 

• Out-of-school time registry. In the fall of 2015, Prime 
Time launched its out-of-school time registry for 
practitioners, built using Salesforce. QIS program 
staff will be required to use the registry to catalogue 
their employment history, as well as participation 

in professional development courses, training and 
conferences. 

• Open access training. Prime Time also offers a series 
of open access training for all OST practitioners. Prime 
Time's training focuses on areas of need that come to 
light across multiple programs. Training that supports 
the needs of both QIS and non-QIS programs 

is offered at various times at multiple locations 
throughout the year. In the first three quarters of 2015, 
771 OST practitioners attended Prime Time training, 
at no cost to OST practitioners working in Palm 
Beach County. In addition, Prime Time is considering 
the development of online training in response to 
practitioner demand. 

Outcomes 

Prime Time is using the Every Hour Counts measurement 
framework to guide its outcome measurement design. Once 
the OST registry is fully operational, Prime Time and the 
Children's Services Council plan to link PBC-PQA program 
data to individual professional development registries, to 
understand better how professional development affects 
the quality ratings of the programs in which staff work. 
Several years of QIS data show that the programs with the 
most dramatic improvement in PBC-PQA scores have two 
key components: 1) directors who encourage their staff to 
participate in professional development offerings specific 
to areas of focus on the PBC- PQA; and 2) staff who engage 
in the improvement planning process by setting program 
goals and putting training skills into practice. 


Prime Time is also using the Devereux Student Strengths 
Assessment to measure development of social/emotional 
skills for youth in out-of-school time. "Eventually we 
will be able to see the continuum of PQA data informing 
professional development that then improves program 
quality and youth outcomes," said Andy McAusland, 
Children's Services Council. 

To get access to youth data reliably and on a continual 
basis, the Children's Services Council funds a staff person 
within the Palm Beach County School District's research 
and development office. The Children's Services Council 
receives school achievement information for the children 
receiving subsidies to attend out-of-school time programs, 
and they plan eventually to match the school data set with 
the Deveraux results to more comprehensively understand 
youth outcomes. 

Key Success Factors 

According to Suzette Harvey, executive director of Prime 
Time Palm Beach County, and Andy McAusland of Children's 
Services Council, the spirit of collaboration, transparent 
communication and a deep level of trust that characterizes 
the relationship between Prime Time and the Children 
Services Council have been key to the success of their efforts 
thus far. In addition, they pointed to the following factors: 

Keeping the QIS voluntary, but offering significant 
incentives for participation. Participation in Prime Time's 
QIS is entirely voluntary, so does not carry the burden of 
a forced mandate for programs. As the QIS has evolved, 
access to important incentives like scholarships and salary 
supplements for staff and to quality advisors for programs 
are restricted to QIS participants, increasing the attraction 
for programs to participate. 

The clear delineation of roles between Prime Time and the 
Children's Services Council. The Children's Services Council 
provides funding to afterschool and summer programs to 
support access for low-income children and to Prime Time 
as the intermediary organization. This frees Prime Time's 
quality advisors to focus solely on quality improvement 
for programs. "Programs can be transparent with us about 
their weaknesses, and we can help them improve. Programs 
know that the quality advisors are not evaluative, but 
focused on connecting them to the resources they need," 
said Harvey. 

Taking the time to create broad stakeholder buy-in 
for each new step. "Prime Time has been strategic in its 
approach to building a professional development system, 
especially given the number, diversity and geographic 
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spread of the practitioners in Palm Beach County," said 
McAusland. "It has methodically implemented each piece, 
making sure there is involvement and support from after- 
school staff and a broad range of others, as well. Prime Time 
moved slowly, crafting each new component as the natural 
next step from the component that preceded it." 

Long-term commitment of the lead funder. Said 
McAusland, "Patience is not always the strongest 
characteristic of funders. But in our relationship with 
Prime Time, we have supported the intentionally slow 
development of the system. We can see where it is going 
- better data will inform us in our efforts to continually 
improve how we develop great staff and improve the quality 
of the programs across the entire system." Prime Time 
also recently received support from Palm Beach County's 
Youth Services Department to support further expansion to 
programs serving middle school-age youth. 

Investment in communications. Prime Time has developed 
a set of communication materials that are designed with 
OST practitioners in mind, to introduce them to professional 
development resources. Trailblazing the Pathways for 
Afterschool: The Route to Afterschool Professional 
Development is a guide to help afterschool practitioners 
navigate the professional development opportunities 
available to them. It provides an overview of the 
educational pathways, scholarship program, and financial 
incentives available. In addition, the guide provides 
step-by-step instructions on how to navigate the Florida 
Department of Children and Families and Palm Beach State 
College websites. 


The Pinkerton Foundation Spearheads Peer-Led 
Quality Improvement in New York City 

Out-of-school time programs serving teenagers face 
unique challenges in improving quality. That's because 
they often are not part of quality improvement systems 
built around child-care licensing requirements or organized 
with resources connected to school-aged child care or 21st 
Century Community Learning Centers. To build capacity and 
improve the quality of these programs, new strategies 
are needed. 

To that end, over the past 13 years, the Pinkerton 
Foundation, a local New York City grantmaker, has catalyzed 
and supported a peer network of high school career 
internship programs, evolving into the go-to resource 
for practitioners and program directors seeking effective 
strategies to improve the quality and impact of their work. 


Career Internships - a definition: Career internship 
programs are designed to expose youth to the world 
of work. These programs aim to help young people 
understand career options, the nature of the workplace 
and how their in- and out-of-school experiences will 
contribute to their work options and choice and further 
education. In his 2013 book: "Youth Education and 
the Role of Society: Rethinking Learning in the FHigh 
School Years," Robert Halpern of the Erikson Institute 
argues that the nation should transform its approach to 
educating adolescents by, among other reforms, vastly 
increasing and improving work-based learning. 

Established in 1966, The Pinkerton Foundation provides 
grants to New York City organizations, with a focus on direct 
service programs that help young people develop the skills, 
self-reliance and strong values necessary to live up to their 
full potential. In the late 1990s, Pinkerton began supporting 
career internship programs as part of its expansion of 
funding for older youth. By the year 2000, Pinkerton was 
supporting 24 internship programs housed in a variety 
of places, such as museums, libraries, hospitals, business 
improvement districts and cultural and social service 
organizations. 

After completing a full round of site visits in 2002, Pinkerton 
Foundation Program Officer Laurie Dien noticed how 
much the group of grantees could learn from one other to 
strengthen each of the programs. She then administered 
a survey to the group to gauge interest in forming a 
network. All 24 programs responded with interest. Prior to 
convening the first meeting, Dien asked each internship 
program to complete a three-page program profile that was 
disseminated to the group beforehand, so that everyone 
would know who else was attending and the basics of 
their program approach. Many had never met the other 
providers. Representatives from all 24 organizations 
attended the first meeting and the Career Internship 
Network (CIN) was on its way. 

Dien found what she hoped for: The diversity of 
participating member organizations fostered an interesting 
cross-pollination of information and resources. Said Dien, 
"Despite the range and unique facets of each internship 
program, members had much to share, becoming important 
resources for one another." 

The next step was to make a modest investment of $15,000 
in the Youth Development Institute (YDI) to coordinate the 
burgeoning network, which included hiring a coordinator 
knowledgeable and passionate about internship programs. 
Established in 1991, it supports the integration of positive 
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youth development principles in programs and systems 
that serve youth, and promotes practices and policies that 
enable young people to thrive. YDI's capacity building 
efforts include staff development, program assessment, 
organizational support, program design, facilitated learning 
communities, tools and information development and 
advancing public policy to provide better support positive 
youth outcomes. 

YDI grounded its work in five research-based principles 
that have been found to be present when youth, especially 
those with significant obstacles in their lives, achieve 
successful adulthood: 1) close relationships with adults; 2) 
high expectations; 3) engaging activities; 4) opportunities 
for contribution; and 5) continuity for adult supports 
over time. Because YDI's approach is distinguished by 
an understanding of and a respect for the complexities 
of youths' lives and the critical role of youth-serving 
organizations in supporting young people, the organization 
was well-positioned to make use of its knowledge, expertise 
and capacity to facilitate the new network. 

Regular meetings first focused on sharing written 
information, including intern orientation and training 
materials, mentor training materials, program evaluation 
and intern assessment forms, and program schedules. 
Continued workshops covered evaluation and assessment, 
youth development principles, intern retention, alumni 
involvement, preventing sexual harassment in the 
workplace and engaging young people in reflective 
activities. Strategies to stay connected with youth alumni 
were shared along with developing a template for an 
alumni-tracking database. Technical assistance is provided 
by CIN's coordinator to individual sites as needed. Lessons 
learned are then shared with the whole network. Site visits 
and training continue to facilitate ongoing learning across 
organizations and build individual skills. 

Dien was successful in bringing the Clark Foundation, and 
later the Altman Foundation, to the table. In addition to 
helping to support the network, Clark provided operational 
funding for internship programs that had reached their time 
limit with Pinkerton through 2010. 

Fast forward 12 years, and the Career Internship Network is 
alive and well. Now supported by the Altman and Pinkerton 
Foundations, CIN has grown to more than 40 member 
programs. CIN's professional development for staff of 
member organizations includes workshops, structured site 
visits, materials, on-site technical assistance and an email 
list where members notify the group about activities other 
program's interns can attend, exchange ideas and program 
advice and build community. From the beginning CIN 


members began offering collaborative programs where 
interns visit each other's programs. The goal is to expand 
interns' knowledge, experiences and career-knowledge 
options. 

Outcomes 

A peer-driven network of programs is a different 
approach from the more-structured development and 
implementation of a QIS. The Pinkerton Foundation 
evaluates CIN by metrics, such as attendance at its 
workshops and meetings, its members' assessments of the 
benefits of CIN, the quality of its workshops and tools and 
the ongoing sustainability and outcomes of its member 
programs. One of the key benefits for member programs is 
that CIN offers a dynamic community and a set of peer- 
developed tools that help mitigate the challenges of staff 
turnover. It also allows for an informal career ladder for staff 
where they move into senior positions at other institutions. 

Annual site visits and written reports spotlight the increase 
in quality of the internship programs that have been 
affected by this shared professional development. YDI 
recently launched a documentation project to create a 
guide to career internships based on the collective learning 
and best practices of the CIN network and to highlight the 
role of learning communities in strengthening program 
quality and impact. 

Key Success Factors 

Providing long-term grantmaker support and advocacy. 

The original idea, early organizing work and seed funding 
for CIN came from the Pinkerton Foundation more than a 
dozen years ago. The foundation has continued to support 
the CIN member programs and the network, and convinced 
other funders to join in supporting both. 

Choosing an intermediary that brings expertise 
and infrastructure to the network. As coordinator of 
the network, YDI leverages not only its infrastructure 
and relationships, but also its deep expertise in youth 
development, high-quality program practice and culture 
that prioritizes the experience of practitioners. 

Hiring a dedicated coordinator with extensive knowledge 
and passion about internship programs. Highlighting best 
practices and helping the group solve challenges were of 
crucial importance. 

Ensuring there are peer-driven priorities and activities. 
From the outset, the network's agenda and activities have 
been driven by the programs to meet their own needs. 
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Grantmakers have maintained flexibility, not requiring that 
the network be required to adopt specific priorities and 
strategies. An advisory committee of YDI representatives 
from member programs and funders helped with the 
network's direction. 

Having a targeted focus. The network's specific focus 
on career internship programs provides strong common 
ground, helps to focus the agenda, and assists the network 
in meeting the needs of member programs. 


Wyoming Afterschool Alliance’s Statewide 
Focus on Quality 

In the fall of 2010, the Wyoming Afterschool Alliance 
(WYAA) decided to step up its focus on quality. To that end, 
it enlisted the National Institute on Out-of-School Time 
(NIOST) at the Wellesley Centers for Women to design and 
implement a statewide OST quality improvement system in 
Wyoming. Central to the effort was the introduction of tools 
allowing programs to analyze data, form plans to address 
problems and measure results. 

As Wyoming's statewide afterschool network, WYAA 
receives core funding from the Charles Stewart Mott 
Foundation. To support the $100,000 cost of the initial pilot, 
WYAA also accessed funding from the state Department 
of Education's 21st Century Community Learning Centers 
program, the Wyoming Community Foundation, the 
Ellbogen Foundation and local private funds. 

Certainly, quality was not a new focus at WYAA. Director 
Linda Barton said, "When I directed an afterschool program, 

I realized how important it is for staff to understand the 
program's mission and goals and what high-quality youth 
work looks like. Clear criteria for hiring staff and high 
expectations had a significant impact on how children 
experienced our program." 

But, Barton continued, "Lots of people think youth work 
is not a real job and many people who work in school 
districts also think this. In order for the OST sector to achieve 
progress toward equity with the school day, we have to 
create organizational structures in our programs. We need 
intentions, plans and implementation built around an 
understanding of what quality is." 

As part of the stepped-up quality effort, in 2011, NIOST 
brought APAS (A Program Assessment System) to Wyoming. 
Director Ellen Gannett explained that NIOST developed 
APAS - which is used by 600+ afterschool programs in 34 


states and Canada - to help build a common approach 
to quality improvement among many different types of 
programs. APAS tools enable programs to analyze program- 
level data, plan interventions and measure the impact. 

"APAS is designed not only to help improve quality across a 
system, but also build the OST field," Gannett said. 

Originally piloted in Massachusetts, APAS includes the 
Survey of Academic and Youth Outcomes (SAYO) and the 
Assessment of Program Practices Tool (APT). The SAYO- 
"Staff & Teacher survey, completed by staff and teachers, 
measures changes in youth skills and outcomes linked to 
long-term healthy development and educational success. 
The SAYO-Youth survey is for youth and includes questions 
about program experiences, sense of competence and 
views of the future. The APT-Observation & Questionnaire 
is used by program staff to examine various aspects of 
program quality. 

For the APAS pilot, 17 sites began to use the SAYO and 
APT, with a focus on measuring student engagement 
in learning. NIOST provided both in-person and virtual 
training to program directors in how to use the new tools, 
and dispatched a cohort of quality advisors - external 
consultants - to coach programs in their ongoing use of the 
tools. Although the external quality advisors model worked 
for NIOST in other places, however, geography proved 
to be a formidable obstacle in Wyoming. Its vast prairies, 
mountains and canyons separate its dozens of small towns, 
making travel a considerably more difficult challenge than 
it is in more populated places. After the first season, NIOST 
and WYAA adjusted the model by training onsite staff to be 
quality advisors. The internal quality advisors took on the 
tasks of observing their own programs and guiding their 
staff teams to develop strategies and activities designed to 
improve quality and student outcomes. 

With NIOST training, program leaders and staff used the APT 
program observation tool to assess their program against 
the tool's quality framework. Then with the assistance of 
quality advisors, the staff reviewed the data and created 
action plans to target those areas that needed to be 
strengthened and reassessed. The action plans were a key 
link to help staff understand the progression from gathering 
and analyzing the data to changing practice based on what 
they learned. The initial action plans revealed to NIOST and 
WYAA the extent of the work before them. 

For example, in one action plan, program staff described 
how they planned to manage disruptive behavior in young 
people through "behavior contracts." WYAA and NIOST 
helped the programs' staff take a different approach, 
instead identifying what it was about their own actions that 
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were potentially creating challenges for young people and 
the changes they needed to make- for example, adding 
prolonged transition times and preventing disruptive 
behavior by adjusting the program and staff practices to 
meet young people's needs more effectively. "We suggested 
replacing behavior contracts with an increase in highly 
engaging, hands-on activities." said Barton. "We don't 
underestimate the difficulty of doing this. In fact, we are 
trying to provide consistent, long term support and build 
internal capacity for programs." 

Outcomes 

Over the past three years, SAYO surveys completed by 
young people participating in APAS programs have shown 
an upward trend in attitudes, skills and behaviors of the 
youth. Quality advisors and site staff report seeing value in 
the quality improvement process and wish to continue their 
APAS work. 

Steve Hamaker, CEO of Greater Wyoming Big Brothers Big 
Sisters, noted, "The APAS system helped our team move 
from being overwhelmed by complex challenges toward 
implementing systematic, data-informed strategies. We 
successfully prioritized and focused improvement efforts 
on manageable, concrete tasks while identifying and 
sustaining program strengths. The support and coaching 
our leadership team receives from NIOST and WYAA are 
invaluable in helping us achieve and celebrate measurable 
gains in quality while creating a culture of continuous 
improvement." 

Programs that participated in all three years of the pilot 
showed increases in engagement in learning from year to 
year and made the largest improvement from fall to spring 
in the third year. NIOST and WYAA found that programs 
involved in the pilot increased their shared vision of quality, 
built a greater understanding of the context of quality 
improvement, teamwork and relationship building, and an 
acceptance of the need to invest time in order to realize 
quality gains. 

Scaling Up 

The initial pilot with NIOST, which involved 17 sites, has 
scaled to include all 35 organizations (74 program sites) that 
receive support from the 21st Century Community Learning 
Center program in Wyoming. The state Department of 
Education budgets $45,000 per year to support WYAA to 
facilitate the process and partner with NIOST. Programs 
are also required to include a line item in their 21st Century 
grant budgets to finance their data reports. 


In addition, programs attend two days of APAS training per 
year, building a professional network of OST colleagues 
across the state. NIOST and WYAA also host monthly 
conference calls with the program staff. The calls provide 
opportunities to ask questions, share successes and ideas 
and, said Barton, "bolster spirits and build community. 

It's important to create a network so people don't feel 
like they are the only ones struggling." WYAA and NIOST 
are developing a peer network to help programs share 
successful ideas and help with challenges. 

Next Step: The Career Development System Framework 

WYAA and NIOST have created the Career Development 
System Framework to establish an over-arching OST 
quality improvement system for the state. Professional 
development is a key component. According to results 
from a 201 1 needs assessment survey administered by 
WYAA to inform the APAS pilot, Wyoming's direct service 
OST staff and leaders want training in multiple areas. 
Informed by the survey, WYAA has sponsored training and 
conferences focused on leadership, project-based learning, 
youth development and STEM. WYAA also recently led the 
development of the state's first afterschool/youth work 
credential program. Launched in May 2015 in partnership 
with a local community college, the online three-semester 
program includes a capstone practicum. WYAA is now 
focused on identifying incentives for staff who enroll in the 
program, raising support for scholarships and developing 
a partnership with the University of Wyoming to offer the 
credential. 

WYAA is also working with the Early Childhood Statewide 
Advisory Council to develop program quality practices for 
programs serving children ages 0-12. 

Key Success Factors 

The initiative's leaders stress the need for personnel, time 
and financial support over time to achieve measurable 
gains in quality. According to a summary report of the 
project written by NIOST, "adopting a data-driven approach 
to quality improvement is a long-term commitment that 
takes at least three years to establish and an ongoing 
commitment (both financial and personnel) to ensure 
continued success." For Wyoming, this commitment 
included the creation of a pilot cohort, successive training 
sessions, repeated data collections at the site level, turning 
data to action at sites, ongoing coaching from NIOST and 
"cheerleading" and financial support from WYAA. 
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Building a Statewide Commitment to High 
Quality Summer Learning in California 

When California's voter-approved Proposition 49 increased 
the state's investment in OST programming by $450 million 
in 2006, the state's 1,000 programs increased by fourfold 
nearly overnight. At the same time, a clear need to focus on 
quality emerged. In response, the David and Lucile Packard 
Foundation joined with other grantmakers to help the 
state's Department of Education create a statewide system 
focused on afterschool quality. 

Not long after, the organization added another component: 
an emphasis on summer learning. When Proposition 49 
was approved, the subject was "not on the radar screen," 
according to Justina Acevedo-Cross, program officer in the 
foundation's Children, Families and Communities program. 
But by 2008, media attention spotlighted new research 
showing the devastating impact of summer learning loss 
on economically disadvantaged children. As a result, the 
foundation expanded its after-school strategy to include 
high-quality opportunities offered over the summer, as 
well. "The foundation's goal is to increase access to year- 
round learning opportunities that support development 
and wellbeing for every child. Our focus was not solely 
on academic programming, or solely on recreation-only 
programs - we wanted to see a blend of the two, with a 
focus on quality," said Acevedo-Cross. 

The Packard Foundation's seven-year initiative (2009-2016) 
has three major components: 

7. Cultivation and Demonstration of High-Quality 

Practice. "When we started, stakeholders did not have a 
common definition or vision of what high-quality summer 
programming looked like," said Acevedo-Cross. The 
foundation selected 10 communities from across the state 
to form a community of practice; each area included the 
school district and district-connected programs run by 
multiple community-based organizations. The communities 
were selected for geographic diversity, existing public 
resources available for summer learning programming 
(including Prop 49 supplemental funds, referred to as ASES, 
or After-School Education and Safety funds), a readiness to 
examine and build their summer learning program using 
quality improvement methods and proximity to a strong 
technical assistance provider. The foundation matched 
each community with coaching and technical assistance 
resources from National Summer Learning Association 
and a local technical assistance provider often connected 
to the state-supported regional hubs for after-school 
technical support. Grants from the foundation supported 
professional development for site leaders and program 


educators to understand the elements they had to build 
into their programs, and how to embed continuous learning 
processes so they could improve quality over time. These 
10 communities developed into what Packard Foundation 
staff refer to as the "showroom communities" for summer 
learning. 

2. Policy Development and Stakeholder Engagement. 

In 2008, summer programming was still not a priority of 
school districts, the state Department of Education or 
state-level policymakers. The Packard Foundation worked 
with advocates to create the statewide Summer Matters 
campaign in 2010, headquartered at the intermediary 
organization Partnership for Children and Youth (PCY) 
and co-chaired by PCY's executive director and the 
state superintendent of public instruction. Summer 
Matters launched a public engagement effort to increase 
understanding among lawmakers, school leaders, teachers, 
parents, youth providers and others about what the 
research showed and why investments in summer learning 
programs were critical for achieving expected student 
outcomes. 

Said Acevedo-Cross: "The 'showroom communities' 
demonstrated high-quality summer learning in action for 
school board members, lawmakers and other decision 
makers - particularly in Sacramento. On site visits, they 
could see the tangible difference between a high-quality, 
engaging summer learning experience and remedial 
summer school. These visits helped build the case for 
investment in summer learning across the state." 

The Summer Matters campaign also explicitly helped 
build expertise among providers, parents and others 
in the showroom communities to speak to the media 
and at meetings and events. They wrote stories about 
summer learning loss in publications read by principals, 
superintendents and school board members. 

Said Acevedo-Cross, "Lots of things happened this year as a 
result of the work that Summer Matters has done. One of the 
major victories was passage of SB 1221 - a state regulation 
giving preference for funding year-round programs with 
the Prop 49 dollars. SB 1221 signals to school leaders, school 
board members, teachers, parents and others that it is 
important to have kids engaged during the summertime." 

SB 1221 also requires programs to demonstrate that they 
are using a quality improvement process to be eligible 
for Prop 49 and 21st Century Community Learning Center 
funding, a requirement that aligns with Packard Foundation- 
supported efforts over the past few years to help create a 
quality-support system for year-round expanded learning 
programming. 
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3. Technical Assistance and System Building to Support 

Quality: In 2006, the California Department of Education 
started building regional hubs for after-school technical 
assistance throughout the state, in an effort to support the 
thousands of new and expanded programs created with 
Prop 49 resources. When Packard's summer strategy was 
introduced in 2009, the approach was to equip the existing 
after-school technical assistance system with expertise 
in supporting high-quality summer programming. "Our 
dollars leveraged the larger state investment and helped 
make the results more robust and accessible to providers," 
said Acevedo-Cross. "Our goal, and we are still trying to 
get there, is that all programs have access to a technical 
assistance provider with expertise in summer learning 
within a two-hour drive." 

In 2012, the Packard Foundation's attention to policymaking 
and ability to find and take advantage of opportunities 
to catalyze large-scale change led the organization to 
invest in the strategic planning process launched by the 
new Afterschool Division of the California Department of 
Education. The Afterschool Division invited a wide range 
of stakeholders to partner with the department in creating 
its strategic plan, aiming to change the focus of the state 
bureaucracy from compliance to quality assurance. With 
support from the Packard Foundation, the S.D. Bechtel, 

Jr. Foundation and other grantmakers, the department 
joined with the California Afterschool Network to create 
multiple working groups that attracted deep and sustained 
participation from diverse stakeholders. Many practitioners 
and advocates on the Summer Matters campaign joined 
the Workgroup on Quality Standards, creating the state's 
first set of quality standards, which were adopted by the 
Department in 2015. 

The strategic planning process is having an impact beyond 
the expanded learning field. "The Afterschool Division 
began breaking new ground for the entire agency with 
a planning process that relied on authentic stakeholder 
engagement - listening to what people and leaders in the 
field had to say. Now this approach has become an exemplar 
for other divisions." said Acevedo-Cross. 

The Packard Foundation will conclude its investment in 
summer learning at the end of 2016. "We didn't go into this 
thinking that by year seven we would be done. But summer 
learning is on the radar of K12 leaders in California in a very 
big way. Investing in summer learning is no longer about 
remedial summer school. School leaders now have choice 
and autonomy about how to spend their funds, and districts 
are seeing the wisdom in investing in quality summer 
learning opportunities for all kids because they are seeing it 
helps them meet their goals," said Acevedo-Cross. 


Outcomes 

The Packard Foundation is measuring the outcomes of its 
initiative in the following ways: 

• Whether districts invest new resources or reallocate 
existing resources to support summer programs. 

And whether K12 leaders better understand and 
more frequently talk about OST and summer learning 
as critical components of children's academic and 
social and emotional learning. A survey conducted 
by an independent evaluator found that two-thirds 
of surveyed California school board members and 
superintendents believe that summer learning loss 

is somewhat of a concern for their district, and 28% 
believe it is a large concern. 

• The extent to which school day and expanded 
learning/summer programs are aligned. Surveys 
by the independent evaluator have found that 
district leaders who express overall support for 
summer learning programs also describe ways that 
the programming is integrated with the goals and 
objectives of their school-day curricula. In addition, 
providers report strong alignment among afterschool 
and summer learning programs - they often use the 
same staff, are led by the same leaders and develop a 
shared understanding of best practices. 

• The quality of the technical assistance provided to 
programs and the extent to which technical assistance 
meets programs' needs. Provider surveys from 2011 
and 2013 show marked improvement, including an 
increase in the percentage of providers rating the 
technical assistance as "excellent" from 44% to 58%. 

In 2013, 80% of afterschool and summer providers 
surveyed reported that the technical assistance they 
tapped met or exceeded their needs. 

In 2017, the Packard Foundation will publish a summary 
of its investment strategy and evaluation, and program 
staff will share their experiences with colleagues and 
stakeholders in a series of meetings planned for 2016-2017. 

Key Success Factors 

Finding strategic investment opportunities within 
the ever-shifting policy environment. Support for the 
Afterschool Division's strategic planning process is one 
example where the Packard Foundation made a relatively 
modest investment that contributed to catalytic results: 
the state's first set of quality standards for after-school 
and summer programs. A second example is the Summer 
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Matters Campaign's focus on ensuring that local school 
leaders understand the importance of summer learning, 
particularly because the state's local control funding 
formula shifts increased budgetary authority to these local 
decision makers. 

Providing sustained support for programs to change 

practice. The Packard Foundation invested in a set of 
communities that represented the diversity of the state, 
and provided ongoing support so they could evolve into 
a strong learning collaborative that would help to define 
what a great summer program looked like. It also sponsored 
technical assistance and two in-person meetings per year. 
Practitioners and technical assistance providers became 
experts at understanding what quality practice entailed and 
how to achieve it. As momentum behind statewide quality 
standards gained steam, this group of practitioners became 
strong champions for the standards. 

Making use of public and private investments. The Packard 
Foundation tapped the regional technical assistance 
leads already in place as part of the California Afterschool 
Division's quality improvement system to be the technical 
assistance providers for its showroom communities as 
well. In addition, multiple local, regional and statewide 
grantmakers made aligned investments in summer learning, 
including Kaiser, the S.D. Bechtel Jr. Foundation, the Noyce 
Foundation, Pacific Gas & Electric Company, Fort Fund, Give 
Something Back, the Lesher Foundation and the Cowell 
Foundation, among others. 

Investing in building the communication and advocacy 
skills of emerging leaders and spokespeople for the 
field. Through the Summer Matters campaign, many 
practitioners found that they were powerful speakers 
about the importance of access to high-quality summer 
learning experiences. Support from the Packard Foundation 
helped build their skills and expertise as communicators 
and contribute their practical expertise to policymaking 
discussions. Said Acevedo-Cross, "It is important for us to 
build emerging leadership in the field of expanded learning. 
People benefit from communications training and learning 
how to develop and disseminate messages. They need skills 
for development, grant writing, organizational capacity, 
sustainability. Investments in this kind of capacity building 
create the potential for long-term impact on youth." 


Increasing Access to High-Quality Summer 
Learning Programs 

Summer camps have long been a focus area for 
philanthropists, especially on the local level. But when 
Karl Alexander and colleagues at Johns Hopkins University 
released research in 2007 showing that over half of the 
academic achievement gap between lower- and higher- 
income youth could be explained by what happens over 
the summer, several leading grantmakers deepened their 
involvement in understanding and supporting increased 
access to high-quality summer learning programs. 

To that end, in 2009, Atlantic Philanthropies seeded the 
National Summer Learning Association (NSLA) as a policy 
and practice organization for the field. NSLA evolved 
from the Center for Summer Learning at Johns Hopkins 
University, which itself had been supported for years 
by the Open Society Foundations (then Institute). NSLA 
engages in policy development and national advocacy 
(for example, National Summer Learning Day) to support 
summer learning. NSLA also sponsors communities of 
practice and develops resources for practitioners, including 
the Comprehensive Assessment of Summer Programs, 
an assessment tool considered appropriate for selecting 
exemplar programs, but too lengthy to be used extensively 
in the field. 

In 2011, the New York-based Wallace Foundation began to 
invest in research and exemplary programs to understand 
effective summer learning approaches. This initiative built 
on Wallace's multiyear investments in city-based afterschool 
systems to expand and improve afterschool programs. One 
practice almost all of the Wallace-supported cities have 
in common is using the Weikart Center's Youth Program 
Quality Assessment (YPQA) tool as an observational 
instrument to assess program quality. The development of 
the YPQA was supported over multiple years by the William 
T. Grant Foundation. In 2012, the Weikart Center and the 
National Summer Learning Association began a joint project 
to create a support system for summer program quality 
improvement that connects directly to afterschool quality 
improvement tools. 

To date, five foundations - William T. Grant Foundation, 

The Wallace Foundation, the David and Lucile Packard 
Foundation, DeVos Foundation and Raikes Foundation - 
have contributed about $560,000 in total to various phases 
and aspects of the work. The results over the past three 
years have represented not only a valuable collaboration 
between two leading national quality improvement-focused 
organizations, but also a unique partnership among both 
national and regional foundations. The Summer Learning 


GRANTMAKERS FOR EDUCATION | QUALITY IN OUT-OF-SCHOOL TIME 



Program Quality Assessment Project has taken shape over 
three major phases of work: 

Phase I - Proof of Concept: Develop and Pilot Summer 
Learning Program Quality Assessment Tool 

In 2013, with the support of the David and Lucile Packard 
Foundation and the William T. Grant Foundation, NSLA 
and the Weikart Center ran a proof-of-concept pilot of a 
quality improvement observational tool specifically geared 
to summer programs and aligned with the PQA. The pilot 
ran in Grand Rapids, Ml, Oakland, CA and Baltimore, MD. 

The purpose of the pilot was to (1) integrate elements of 
existing Weikart Center and NSLA tools as a scalable quality 
improvement intervention for use in summer learning 
programs, and (2) to implement this process in a small 
number of programs in order to gain a better understanding 
of how it worked. Findings from the pilot revealed key areas 
for improvement, including streamlining the tool, revising 
the data collection protocol and developing better supports 
to enable sites to use the data for program improvement. 

Phase II - Feasibility: Develop and Pilot Summer Learning 
Program Quality Intervention Process 

In summer 2014, Weikart and NSLA began testing whether it 
was feasible for communities to launch the assessment and 
accompanying training at a larger scale than the previous 
year. The Wallace Foundation worked with a set of regional 
funders - Raikes Foundation (Seattle, WA), the David and 
Lucile Packard Foundation (Oakland CA and Stockton, CA) 
and the DeVos Foundation (Grand Rapids, Ml) - to support 
the feasibility study. The Wallace Foundation funded the 
Weikart Center and NSLA to refine the Summer Learning 
PQA and provide training and technical assistance to 
sites, while each of the other foundations supported the 
pilot sites in their regions. The feasibility study found that 
programs had a high satisfaction rate with the Summer 
Learning PQA, but were more successful using it if a quality 
improvement infrastructure was already in place. Cities 
that had some familiarity with the YPQA process and a pool 
of trained assessors successfully integrated the Summer 
Learning PQA into their activities. The pilot also found great 
variety in the quality of the summer programs within and 
across cities. 


Phase III — Scaling up: Refine and Expand Summer 
Learning PQA Process 

In the next phase, The Wallace Foundation, Weikart Center 
and NSLA selected Denver and St. Paul, two places with city- 
wide afterschool systems and mature quality improvement 
processes , to implement the Summer PQA; that meant 
reaching 30 program sites in summer 2015 and 60 sites 
in 2016. In addition, the Packard Foundation is providing 
support to the Weikart Center and NLSA to refine their 
observation, interview and data management tools, and 
the Raikes Foundation is funding the Weikart Center's work 
with the statewide intermediary, School's Out Washington 
and the Seattle Public Schools. "The strong partnership 
between the school district and our statewide intermediary 
gave us the opportunity to look at the relationship between 
highquality summer programs as measured by the SLPQA 
and youth outcomes. While we are still learning what 
works in which settings, preliminary results demonstrate 
the promise of highquality programs," said Juliet Taylor, 
program officer at the Raikes Foundation. 

Key Success Factors 

Leverage prior investments. The stakeholders in 
this project intentionally used their own and other 
philanthropies' prior investments to create and test a 
validated tool that could be used at scale throughout the 
country to build quality in summer programs. 

Respect grantmakers priorities and constraints. The 

project was designed so each grantmaker could invest in 
alignment with its own strategies. All the investments built 
upon one another. 

Set clear expectations for all partners. The project 
partners committed to spending the time to communicate 
with each other consistently, hold quarterly calls among 
the national and regional funding partners and maintain 
close contact with the grantees. The project was designed 
with clear roles, goals and expectations for all stakeholders, 
funders included. The sites participating in the pilot phase 
were assured that the quality data emerging from their pilot 
of the Summer PQA would not be used for evaluation, but to 
inform quality improvement. 

Support intermediaries as key drivers of quality. The 
project was most successful in the places that had an 
experienced and trusted local intermediary with the ability 
to carry forward the day-to-day work with providers. 
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Principles for Effective 
Education Grantmaking 




Discipline and Focus 


In education, where public dollars dwarf private investments, a funder 
has greater impact when grantmaking is carefully planned and targeted. 


Information, ideas and advice from diverse sources, as well as openness 
to criticism and feedback, can help a funder make wise choices. 


Resources Linked to Results 


logic-driven "theory of change” helps a grantmaker think clearly 
about how specific actions will lead to desired outcomes, thus linking 
resources with results. 




Effective Grantees 


A grantmaker is effective only when its grantees are effective. 
Especially in education, schools and systems lack capacity and grantees 
(both inside and outside the system) may require deeper support. 


A funder succeeds by actively engaging its partners - the individuals, 
institutions and communities connected with an issue - to ensure 
"ownership" of education problems and their solutions. 


principle 

9 


Leverage, Influence and Collaboration 


The depth and range of problems in education make it difficult to achieve 
meaningful change in isolation or by funding programs without changing 
public policies or opinions. A grantmaker is more effective when working 
with others to mobilize and deploy as many resources as possible in order 
to advance solutions. 


principle 

no. 


The most important problems in education are often the most complex 
and intractable, and will take time to solve. 


principle 

no. 



Innovation and Constant Learning 


Even while acting on the best available information - as in Principle #2 - a 
grantmaker can create new knowledge about ways to promote educational 
success. Tracking outcomes, understanding costs and identifying what 
works-and what doesn't-are essential to helping grantmakers and their 
partners achieve results. 


@ 
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